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Abstract 



Survey propagation is a powerful technique from statistical physics that 
has been applied to solve the 3-SAT problem both in principle and in 
practice. We give, using only probability arguments, a common deriva- 
tion of survey propagation, belief propagation and several interesting hy- 
CN ■ brid methods. We then present numerical experiments which use WSAT 

(a widely used random-walk based SAT solver) to quantify the complex- 
ity of the 3-SAT formulae as a function of their parameters, both as ran- 
domly generated and after simplification, guided by survey propagation. 
^Q I Some properties of WSAT which have not previously been reported make 

(<— ^ i it an ideal tool for this purpose - its mean cost is proportional to the num- 

T^lj- ' ber of variables in the formula (at a fixed ratio of clauses to variables) in 

f^ I the easy-SAT regime and slightly beyond, and its behavior in the hard- 

SAT regime appears to reflect the underlying structure of the solution 
C^ ' space that has been predicted by replica symmetry-breaking arguments. 

An analysis of the tradeoffs between the various methods of search for 
satisfying assignments shows WSAT to be far more powerful than has 
' O I been appreciated, and suggests some interesting new directions for prac- 

G . tical algorithm development. 

O 



1 Introduction 



Random 3-SAT is a classic problem in combinatorics, at the heart of computational com- 
j^ ' plexity studies and a favorite testing ground for both exactly analyzable and heuristic so- 

lution methods which are then applied to a wide variety of problems in machine learning 
and artificial intelligence. It consists of a ensemble of randomly generated logical expres- 
sions, each depending on N Boolean variables Xi, and constructed by taking the AND of 
M clauses. Each clause a consists of the OR of 3 "literals" i/i a . j/i q is taken to be either Xi 



or -iXi at random with equal probability, and the three values of the index i in each clause 
are distinct. Conversely, the neighborhood of a variable Xi is Vi, the set of all clauses in 
which Xi or -ix^ appear. For each such random formula, one asks whether there is some 
set of Xi values for which the formula evaluates to be TRUE. The ratio a = M/N controls 
the difficulty of this decision problem, and predicts the answer with high accuracy, at least 
as both N and M tend to infinity, with their ratio held constant. At small a, solutions are 
easily found, while for sufficiently large a there are almost certainly no satisfying configu- 
rations of the Xi, and compact proofs of this fact can be constructed. Between these limits 
lies a complex, spin-glass-like phase transition, at which the cost of analyzing the problem 
with either exact or heuristic methods explodes. 

A recent series of papers drawing upon the statistical mechanics of disordered materials 
has not only clarified the nature of this transition, but also lead to a thousand-fold increase 
in the size of the concrete problems that can be solved [1, 2, 3] This paper provides a 
derivation of the new methods using nothing more complex than probabilities, suggests 
some generalizations, and reports numerical experiments that disentangle the contributions 
of the several component heuristics employed. For two related discussions, see [4, 5]. 

An iterative "belief propagation" [6] (BP) algorithm for K-S AT can be derived to evaluate 
the probability, or "belief," that a variable will take the value TRUE in variable config- 
urations that satisfy the formula considered. To calculate this, we first define a message 
("transport") sent from a variable to a clause: 

• ti^a is the probability that variable Xi satisfies clause a 

In the other direction, we define a message ("influence") sent from a clause to a variable: 

• ia^i is the probability that clause a is satisfied by another variable than Xi 

In 3-SAT, where clause a depends on variables Xi, Xj and Xk, BP gives the following 
iterative update equation for its influence. 

Ai) _ Ai) ,M _Ai) Ai) n\ 

The BP update equations for the transport ti^a involve the products of influences acting 
on a variable from the clauses which surround Xi, forming its "cavity," Vi, sorted by which 
literal (xi or -^Xi) appears in the clause: 

4'= n *&-» (2) 
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The superscripts {I) and (/ — 1) denote iteration. The probabilistic interpretation is the 
following: suppose we have il_^i for all clauses 6 connected to variable i. Each of these 
clauses can either be satisfied by another variable (with probability i\,_^i), or not be satisfied 
by another variable (with probability ( 1 — ij^l^i ) ), and also be satisfied by variable i itself. 



If we set variable Xi to 0, then some clauses are satisfied by Xi, and some have to be satisfied 
by other variables. The probability that they are all san's/zet/ is Jlfca^a Vi ,,=xi H~*i- Similarly, 



if Xi is set to 1 then all these clauses b are satisfied with probability Jlfcaia v =^x- A~*i- 
The products in (3) can therefore be interpreted as joint probabilities of independent events. 
Variable Xi can be or 1 in a solution if the clauses in which Xi appears are either satisfied 
directly by Xi itself, or by other variables. Hence 

Prob(xO= ^o^'^i and Prob(^xO= ^o^'^i W 

A BP-based decimation scheme results from fixing the variables with largest probability to 
be either true or false. We then recalculate the beliefs for the reduced formula, and repeat. 

To arrive at SP we introduce a modified system of beliefs: every variable falls into one of 
three classes: TRUE in all solutions (1); FALSE in all solutions(O); and TRUE in some and 
FALSE in other solutions (free). The message from a clause to a variable (an influence) 
is then the same as in BP above. Although we will again only need to keep track of one 
message from a variable to a clause (a transport), it is convenient to first introduce three 
ancillary messages: 

• 7i^a(l) is the probability that variable Xi is true in clause a in all solutions 

• Ti^ai^) is the probability that variable xi is false in clause a in all solutions 

• Ti^aif'ree) is the probability that variable Xi is true in clause a in some solutions 
and false in others. 

Note that there are here three transports for each directed link i ^ a, from a variable to 
a clause, in the graph. As in BP, these numbers will be functions of the influences from 
clauses to variables in the preceeding update step. Taking again the incoming influences 
independent, we have 

flLifree) cc U.eVAa ^^ 

f^'lM + T?Ulree) ex n.eyAa... .=.. ^^ (5) 

fl^aW + T^^aUree) (x n.eyAa,..,.=^.. C'^ 

The proportionality indicates that the probabilities are to be normalized. We see that the 
structure is quite similar to that in BP. But we can make it closer still by introducing ti^a 
with the same meaning as in BP. In SP it will then, as the case might be, be equal to to 

T.^aifree) + T,^„(0) or T.^aijree) + T,^„(l). That gives (compare (3)): 
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The update equations for ti^a are the same in SP as in BP, i.e. one uses (1) in SP as well. 
Similarly to (4), decimation now removes the most fixed variable, i.e. the one with the 
largest absolute value of {A° - A})/{A^ + A} - AjA'^). Given the complexity of the 
original derivation of SP [1, 2], it is remarkable that the SP scheme can be interpreted as 
a type of belief propagation in another belief system. And even more remarkable that the 
final iteration formulae differ so little. 

A modification of SP which we will consider in the following is to interpolate between BP 
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Figure 1 ; Dependence of decimation depth on the interpolation parameter p. 



{p = 0) and SP (p = 1) ' by considering equations 
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We do not have an interpretation of the intermediate cases of p as behef systems. 



2 The Phase Diagram of 3-SAT 

Early work on developing 3-SAT heuristics discovered that as a is increased, the problem 
changes from being easy to solve to extremely hard, then again relatively easy when the 
formulae are almost certainly UNSAT. It was natural to expect that a sharp phase boundary 
between SAT and UNSAT phases in the limit of large N accompanies this "easy-hard-easy" 
observed transition, and the finite-size scaling results of [7] confirmed this. Their work 
placed the transition at about a = 4.2. Monasson and Zecchina [8] soon showed, using the 
replica method from statistical mechanics, that the phase transition to be expected had un- 
usual characteristics, including "frozen variables" and a highly nonuniform distribution of 
solutions, making search difficult. Recent technical advances have made it possible to use 
simpler cavity mean field methods to pinpoint the SAT/UNSAT boundary at a = 4.267 and 
suggest that the "hard-SAT" region in which the solution space becomes inhomogeneous 
begins at about a — 3.92. These calculations also predicted a specific solution structure 
(termed 1-RSB for "one step replica symmetry-breaking") [1, 2] in which the satisfiable 
configurations occur in large clusters, maximally separated from each other. Two types 
of frozen variables are predicted, one set which take the same value in all clusters and a 
second set whose value is fixed within a particular cluster The remaining variables are 
"paramagnetic" and can take either value in some of the states of a given cluster. A careful 
analysis of the 1-RSB solution has subsequently shown that this extreme structure is only 
stable above a — 4.15. Between 3.92 and 4.15 a wider range of cluster sizes, and wide 
range of inter-cluster Hamming distances are expected[9]. As a result, we expect the val- 
ues a = 3.9, 4.15 and 4.267 to separate regions in which the nature of the 3-SAT decision 
problem is distinctly different. 



This interpolation has also been considered and implemented by R. Zecchina and co-workers. 



"Survey-induced decimation" consists of using SP to determine the variable most likely to 
be frozen, then setting that variable to the indicated frozen value, simplifying the formula 
as a result, updating the SP calculation, and repeating the process. For a < 3.9 we expect 
SP to discover that all spins are free to take on more than one value in some ground state, 
so no spins will be decimated. Above 3.9, SP ideally should identify frozen spins until 
all that remain are paramagnetic. The depth of decimation, or fraction of spins reminaing 
when SP sees only paramagnetic spins, is thus an important characteristic. We show in 
Fig. 1 the fraction of spins remaining after survey-induced decimation for values of a from 
3.85 to 4.35 in hundreds of formulae with iV — 10, 000. The error bars show the standard 
deviation, which becomes quite large for large values of a. To the left of a = 4.2, on the 
descending part of the curves, SP reaches a paramagnetic state and halts. On the right, or 
ascending portion of the curves, SP stops by simply failing to converge. 

Fig 1 also shows how different the behavior of BP and the hybrids between BP and SP are 
in their decimation behavior. We studied BP (p = 0), underrelaxed SP (p — 0.95), SP, and 
overrelaxed SP (p = 1.05). BP and underrelaxed SP do not reach a paramagnetic state, but 
continue until the formula breaks apart into clauses that have no variables shared between 
them. We see in Fig. 1 that BP stops working at roughly a = 3.9, the point at which SP 
begins to operate. The underrelaxed SP behaves like BP, but can be used well into the RSB 
region. On the rising parts of all four curves in Fig 1, the scheme halted as the surveys 
ceased to converge. Overrelaxed SP in Fig. 1 may give reasonable recommendations for 
simplification even on formulae which are likely to be UNSAT 

3 Some Background on WSAT 

Next we consider WSAT, the random walk-based search routine used to finish the job of 
exhibiting a satisfying configuration after SP (or some other decimation advisor) has sim- 
plified the formula. The surprising power exhibited by SP has to some extent obscured the 
fact that WSAT is itself a very powerful tool for solving constraint satisfaction problems, 
and has been widely used for this. Its running time, expressed in the number of walk steps 
required for a successful search is also useful as an informal definition of the complexity of 
a logical formula. Its history goes back to Papadimitriou's [10] observation that a subtly bi- 
ased random walk would with high probability discover satisfying solutions in the simpler 
2-SAT problem after, at worst, O(iV^) steps. His procedure was to start with an arbitary 
assignment of values to the binary variables, then reverse the sign of one variable at a time 
using the following random process: 

• select an unsatisfied clause at random 

• select at random a variable that appears in the clause 

• reverse that variable 

This procedure, sometimes called RWalkSAT, works because changing the sign of a vari- 
able in an unsatisfied clause always satisfies that clause and, at first, has no net effect 
on other clauses. It is much more powerful than was proven initially. Two recent pa- 
pers [12, 13]. have argued analytically and shown experimentally that Rwalksat finds sat- 
isfying configurations of the variables after a number of steps that is proportional to A^ for 
values of a up to roughly 2.7, after which this cost increases exponentially with N. 

The second trick in WSAT was introduced by Kautz and Selman [11]. They also choose 
an unsatisfied clause at random, but then reverse one of the "best" variables, selected at 
random, where "best" is defined as causing the fewest satisfied clauses to become unsatis- 
fied. For robustness, they mix this greedy move with random moves as used in RWalkSAT, 
recommending an equal mixture of the two types of moves. Barthel et a/. [13] used these 
two moves in numerical experiments, but found little improvement over RWalkSAT. 
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Figure 2: (a) Median of WSAT cost per variable in 3-SAT as a function of a. (b) Variance 
of WSAT cost, scaled by N. 



There is a third trick in the most often used variant of WSAT, introduced slightly later [14]. 
If any variable in the selected unsatisfied clause can be reversed without causing any other 
clauses to become unsatisfied, this "free" move is immediately accepted and no further 
exploration is required. Since we shall show that WSAT works well above a — 2.7, this 
third move apparently gives WSAT its extra power. Although these moves were chosen 
by the authors of WSAT after considerable experiment, we have no insight into why they 
should be the best choices. 

In Fig. 2a, we show the median number of random walk steps per variable taken by the 
standard version of WSAT to solve 3-SAT formulas at values of a ranging from 0.5 to 4.3 
and for formulae of sizes ranging from N = 1000 to A^ = 20000. The cost of WSAT 
remains linear in N well above a = 3.9. WSAT cost distributions were collected on at 
least 1000 cases at each point. Since the distributions are asymmetric, with strong tails 
extending to higher cost, it is not obvious that WSAT cost is, in the statistical mechanics 
language, self-averaging, or concentrated about a well-defined mean value which domi- 
nates the distribution as iV ^ oo. To test this, we calculated higher moments of the WSAT 
cost distribution and found that they scale with simple powers of N. For example, in Fig. 
2b, we show that the variance of the WSAT cost per variable, scaled up by N, is a well- 
defined function of a up to almost 4.2. The third and fourth moments of the distribution 
(not shown) also are constant when multiplied by A^ and by iV^, respectively. The WSAT 
cost per variable is thus given by a distribution which concentrates with increasing N in 
exactly the way that a process governed by the usual laws of large numbers is expected to 
behave, even though the typical cost increases by six orders of magnitude as we move from 
the trivial cases to the critical regime. 

A detailed analysis of the cost distributions which we observed will be published elsewhere 
but we conclude that the median cost of solving 3-SAT using the WSAT random walk 
search, as well as the mean cost if that is well-defined, remains linear in A^ up to a — 
4.15, coincidentally the onset of 1-RSB. In the 1-RSB regime, the WSAT cost per variable 
distributions shift to higher values as N increases, and an exponential increase in cost 
with N is likely. Is 4.15 really the endpoint for WSAT's linearity, or will the search cost 
per variable converge at still larger values of N which we could not study? We define a 
rough estimate of N onset {o) by study of the cumulative distributions of WSAT cost as the 
value of N for a given a above which the distributions cross at a fixed percentile. Plotting 
\og{Nonset) against log(4.15 — a) in Fig. 3, we find strong indication that 4.15 is indeed 
an asymptote for WSAT. 
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Figure 3: Size N at which WSAT cost is linear in N as function of 4.15 — a. 
Figure 4: WSAT cost, before and after SP-guided decimation. 



4 Practical Aspects of SP + WSAT 



The power of SP comes from its use to guide decimation by identifying spins which can 
be frozen while minimally reducing the number of solutions that can be constructed. To 
assess the complexity of the reduced formulae that decimation guided in this way produces 
we compare, in Fig. 4, the median number of WSAT steps required to find a satisfying 
configuration of the variables before and after decimation. To a rough approximation, we 
can say that SP caps the cost of finding a solution to what it would be at the entry to the 
critical regime. There are two factors, the reduction in the number of variables that have 
to be searched, and the reduction of the distance the random walk must traverse when it is 
restricted to a single cluster of solutions. In Fig. 2c the solid lines show the WSAT costs 
divided by N, the original number of variables in each formula. If we instead divide the 
WSAT cost after decimation by the number of variables remaining, the complexity measure 
that we obtain is only a factor of two larger, as shown by the dotted lines. The relative cost 
of running WSAT without benefit of decimation is 3-4 decades larger 

We measured the actual compute time consumed in survey propagation and in WSAT. For 
this we used the Zecchina group's version 1.3 survey propagation code, and the copy of 
WSAT (H. Kautz's release 35, see [15]) that they have also employed. All programs were 
run on a Pentium IV Xeon 3GHz dual processor server with 4GB of memory, and only one 
processor busy. We compare timings from runs on the same 100 formulas with A^ = 10000 
and a = 4.1 and 4.2 (the formulas are simply extended slightly for the second case). In the 
first case, the 100 formulas were solved using WSAT alone in 921 seconds. Using SP to 
guide decimation one variable at a time, with the survey updates performed locally around 
each modified variable, the same 100 formulas required 6218 seconds to solve, of which 
only 3 1 sec was spent in WSAT. 

When we increase alpha to 4.2, the situation is reversed. Running WSAT on 100 formulas 
with N = 10000 required 27771 seconds on the same servers, and would have taken 
even longer if about half of the runs had not been stopped by a cutoff without producing a 
satisfying configuration. In contrast, the same 100 formulas were solved by SP followed 
with WSAT in 10,420 sec, of which only 300 seconds were spent in WSAT. The cost of 
SP does not scale linearly with N, but appears to scale as 7V^ in this regime. We solved 
100 formulas with N = 20, 000 using SP followed by WSAT in 39643 seconds, of which 
608 sec was spent in WSAT. The cost of running SP to decimate roughly half the spins has 
quadrupled, while the cost of the final WSAT runs remained proportional to A''. 

Decimation must stop short of the paramagnetic state at the highest values of a, to avoid 
having SP fail to converge. In those cases we found that WSAT could sometimes find 



satisfying configurations if started slightly before this point. We also explored partial deci- 
mation as a means of reducing the cost of WS AT just below the 1-RSB regime, but found 
that decimation of small fractions of the variables caused the WSAT running times to be 
highly unpredictable, in many cases increasing strongly. As a result, partial decimation 
does not seem to be a useful approach. 

5 Conclusions and future work 

The SP and related algorithms are quite new, so programming improvements may modify 
the practical conclusions of the previous section. However, a more immediate target for 
future work could be the WSAT algorithms. Further directing its random choices to incor- 
porate the insights gained from BP and SP might make it an effective algorithm even closer 
to the SAT/UNSAT transition. 
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